Dynamic work scheduling in FileStream by alamb · Pull Request #21351 · apache/datafusion

alamb · 2026-04-03T16:53:51Z

Stacked on

Which issue does this PR close?

part of Introduce morsel-driven Parquet scan #20529
Closes Sketch out a Morselize API #20820

Rationale for this change

The whole point of this sequence of PRs is to enable dynamic work scheduling in the FileStream (so that if a task is done it can look at any remaining work)

What changes are included in this PR?

Add shared state across sibling FileStream's
Sibling streams put their file work into a shared queue when it can be reordered

Note there are a bunch of other things that are NOT included in this PR, including

Trying to limit concurrent IO (this PR has the same properties as main -- up to one outstanding IO per partition)
Trying to issue multiple IOs by the same partition (aka to interleave IO and CPU work)
Splitting files into smaller units (e.g. across row groups)

Are these changes tested?

Yes by existing functional and benchmark tests, as well as new functional tests

Are there any user-facing changes?

Yes, faster performance (see benchmarks0

Dandandan · 2026-04-06T12:14:20Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and alamb_reschedule_io
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃                   alamb_reschedule_io ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.30 / 4.56 ±6.37 / 17.29 ms │          1.31 / 4.55 ±6.35 / 17.25 ms │     no change │
│ QQuery 1  │        14.65 / 15.05 ±0.29 / 15.56 ms │        12.00 / 12.42 ±0.27 / 12.83 ms │ +1.21x faster │
│ QQuery 2  │        44.23 / 44.61 ±0.42 / 45.39 ms │        35.97 / 36.54 ±0.31 / 36.90 ms │ +1.22x faster │
│ QQuery 3  │        43.10 / 44.46 ±1.35 / 46.69 ms │        34.41 / 35.17 ±0.70 / 36.12 ms │ +1.26x faster │
│ QQuery 4  │     293.78 / 305.01 ±7.65 / 316.85 ms │     260.16 / 263.55 ±4.78 / 272.96 ms │ +1.16x faster │
│ QQuery 5  │     348.16 / 352.47 ±4.23 / 358.82 ms │     302.86 / 308.72 ±5.56 / 319.27 ms │ +1.14x faster │
│ QQuery 6  │           5.80 / 6.57 ±0.63 / 7.63 ms │           6.62 / 7.86 ±0.78 / 8.75 ms │  1.20x slower │
│ QQuery 7  │        17.11 / 17.74 ±0.66 / 18.58 ms │        13.90 / 14.35 ±0.39 / 14.93 ms │ +1.24x faster │
│ QQuery 8  │     420.14 / 429.93 ±8.45 / 444.77 ms │    378.42 / 390.37 ±11.88 / 408.19 ms │ +1.10x faster │
│ QQuery 9  │     663.03 / 672.84 ±5.72 / 680.54 ms │    521.39 / 531.97 ±10.80 / 552.34 ms │ +1.26x faster │
│ QQuery 10 │        92.14 / 94.93 ±2.52 / 99.25 ms │        76.47 / 77.79 ±1.46 / 80.45 ms │ +1.22x faster │
│ QQuery 11 │     104.15 / 105.38 ±0.69 / 106.30 ms │        86.41 / 87.26 ±0.97 / 89.08 ms │ +1.21x faster │
│ QQuery 12 │     357.65 / 366.42 ±6.72 / 373.88 ms │     297.51 / 304.37 ±4.38 / 309.09 ms │ +1.20x faster │
│ QQuery 13 │     458.02 / 470.34 ±9.64 / 480.11 ms │    428.91 / 443.12 ±10.92 / 458.50 ms │ +1.06x faster │
│ QQuery 14 │     354.42 / 359.89 ±4.07 / 367.09 ms │     305.02 / 314.80 ±6.65 / 325.44 ms │ +1.14x faster │
│ QQuery 15 │    370.27 / 382.31 ±15.57 / 412.33 ms │    332.09 / 345.10 ±15.11 / 373.91 ms │ +1.11x faster │
│ QQuery 16 │    734.79 / 751.32 ±12.64 / 771.68 ms │    690.14 / 723.75 ±56.61 / 836.78 ms │     no change │
│ QQuery 17 │    721.23 / 731.36 ±10.62 / 751.20 ms │    686.92 / 718.61 ±57.67 / 833.81 ms │     no change │
│ QQuery 18 │ 1418.87 / 1491.07 ±39.56 / 1534.75 ms │ 1378.18 / 1409.22 ±49.85 / 1508.29 ms │ +1.06x faster │
│ QQuery 19 │        36.52 / 38.08 ±1.80 / 40.51 ms │      32.21 / 62.83 ±37.79 / 137.37 ms │  1.65x slower │
│ QQuery 20 │    723.14 / 740.27 ±16.11 / 760.99 ms │    513.72 / 527.72 ±10.71 / 544.27 ms │ +1.40x faster │
│ QQuery 21 │     767.80 / 774.94 ±6.76 / 787.66 ms │     597.68 / 606.11 ±7.93 / 618.51 ms │ +1.28x faster │
│ QQuery 22 │  1131.51 / 1137.57 ±7.04 / 1151.24 ms │ 1062.84 / 1072.76 ±10.36 / 1091.17 ms │ +1.06x faster │
│ QQuery 23 │ 3068.54 / 3099.60 ±19.20 / 3116.41 ms │ 3302.70 / 3323.50 ±16.22 / 3352.12 ms │  1.07x slower │
│ QQuery 24 │     101.01 / 104.87 ±2.81 / 109.76 ms │        43.34 / 46.80 ±3.25 / 52.18 ms │ +2.24x faster │
│ QQuery 25 │     140.64 / 141.29 ±0.67 / 142.40 ms │     114.94 / 116.41 ±1.24 / 118.08 ms │ +1.21x faster │
│ QQuery 26 │     102.97 / 104.39 ±0.80 / 105.19 ms │        43.89 / 45.52 ±1.57 / 47.70 ms │ +2.29x faster │
│ QQuery 27 │     852.42 / 859.56 ±7.14 / 871.93 ms │     673.53 / 683.34 ±5.19 / 688.03 ms │ +1.26x faster │
│ QQuery 28 │ 7750.35 / 7784.12 ±20.30 / 7804.81 ms │ 6987.03 / 7070.54 ±44.85 / 7114.20 ms │ +1.10x faster │
│ QQuery 29 │        52.98 / 55.46 ±4.12 / 63.62 ms │       43.51 / 55.84 ±19.75 / 95.00 ms │     no change │
│ QQuery 30 │     364.88 / 374.97 ±7.27 / 384.57 ms │    332.34 / 344.98 ±13.87 / 371.55 ms │ +1.09x faster │
│ QQuery 31 │    372.94 / 390.15 ±14.22 / 416.32 ms │    335.38 / 350.66 ±13.06 / 374.32 ms │ +1.11x faster │
│ QQuery 32 │ 1046.43 / 1088.88 ±41.29 / 1149.35 ms │ 1038.82 / 1060.23 ±30.52 / 1120.27 ms │     no change │
│ QQuery 33 │  1488.55 / 1499.96 ±7.93 / 1512.60 ms │  1459.04 / 1470.59 ±7.68 / 1480.30 ms │     no change │
│ QQuery 34 │ 1488.46 / 1500.84 ±11.87 / 1519.15 ms │ 1447.16 / 1472.97 ±18.26 / 1500.69 ms │     no change │
│ QQuery 35 │     402.87 / 405.09 ±2.05 / 408.76 ms │     322.39 / 325.27 ±4.63 / 334.50 ms │ +1.25x faster │
│ QQuery 36 │     117.00 / 122.22 ±2.88 / 125.83 ms │        70.69 / 72.64 ±1.35 / 74.20 ms │ +1.68x faster │
│ QQuery 37 │        50.38 / 51.37 ±0.73 / 52.43 ms │        38.61 / 39.88 ±0.92 / 41.10 ms │ +1.29x faster │
│ QQuery 38 │        76.42 / 77.64 ±0.93 / 78.65 ms │        40.72 / 42.41 ±1.03 / 43.59 ms │ +1.83x faster │
│ QQuery 39 │     222.54 / 226.06 ±2.44 / 229.01 ms │     130.19 / 134.34 ±3.95 / 140.15 ms │ +1.68x faster │
│ QQuery 40 │        22.89 / 26.76 ±2.41 / 30.20 ms │        17.75 / 20.33 ±2.11 / 22.87 ms │ +1.32x faster │
│ QQuery 41 │        19.87 / 21.12 ±0.68 / 21.88 ms │        15.20 / 15.73 ±0.36 / 16.07 ms │ +1.34x faster │
│ QQuery 42 │        20.10 / 20.26 ±0.15 / 20.52 ms │        13.64 / 14.18 ±0.41 / 14.70 ms │ +1.43x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                  │ 27291.72ms │
│ Total Time (alamb_reschedule_io)   │ 25005.09ms │
│ Average Time (HEAD)                │   634.69ms │
│ Average Time (alamb_reschedule_io) │   581.51ms │
│ Queries Faster                     │         33 │
│ Queries Slower                     │          3 │
│ Queries with No Change             │          7 │
│ Queries with Failure               │          0 │
└────────────────────────────────────┴────────────┘

Resource Usage
clickbench_partitioned — base (merge-base)

Metric Value
Wall time 137.6s
Peak memory 40.4 GiB
Avg memory 31.6 GiB
CPU user 1299.0s
CPU sys 87.2s
Peak spill 0 B
clickbench_partitioned — branch

Metric Value
Wall time 126.2s
Peak memory 41.5 GiB
Avg memory 28.2 GiB
CPU user 1313.5s
CPU sys 87.9s
Peak spill 0 B
File an issue against this benchmark runner

It's happening!

alamb · 2026-04-06T12:45:11Z

It's happening!

Yes! Finally -- and no regressions this time!

I have a pile of stacked PRs:

Now that I have some evidence that this PR make them faster, I'll go back and get them ready to review

alamb · 2026-04-06T13:10:29Z

…er` (#21327) ~(Draft until I am sure I can use this API to make FileStream behave better)~ ## Which issue does this PR close? - part of #20529 - Needed for #21351 - Broken out of #20820 - Closes #21427 ## Rationale for this change I can get 10% faster on many ClickBench queries by reordeirng files at runtime. You can see it all working together here: #21351 To do do, I need to rework the FileStream so that it can reorder operations at runtime. Eventually that will include both CPU and IO. This PR is a step in the direction by introducing the main Morsel API and implementing it for Parquet. The next PR (#21342) rewrites FileStream in terms of the Morsel API ## What changes are included in this PR? 1. Add proposed `Morsel` API 2. Rewrite Parquet opener in terms of that API 3. Add an adapter layer (back to FileOpener, so I don't have to rewrite FileStream in the same PR) My next PR will rewrite the FileStream to use the Morsel API ## Are these changes tested? Yes by existing CI. I will work on adding additional tests for just Parquet opener in a follow on PR ## Are there any user-facing changes? No

Dandandan · 2026-04-10T08:52:28Z

│ Average Time (HEAD)                │   2453.33ms │
│ Average Time (alamb_reschedule_io) │   2074.70ms │

I think this is pretty nice - for high latency stores, the relative improvement is (much) larger as it benefits evenly balanced queries more as well.

alamb · 2026-04-10T15:20:05Z

I am still iterating / futzing with the APIs / how to plumb this, but I think I mostly have a proposal ready (will push soon)

I'll rebase / mark it for review once this one goes in:

Rewrite FileStream in terms of Morsel API #21342

This reverts commit c479ed68e02a4337eb9cbafa8ec1d70b4e5c2345.

alamb · 2026-04-10T15:54:23Z

run benchmarks clickbench_partitioned

env:
    SIMULATE_LATENCY: true

alamb · 2026-04-10T15:54:33Z

run benchmark clickbench_partitioned

adriangbot · 2026-04-10T15:57:06Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4225016775-1054-68nwr 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing alamb/reschedule_io (9124f2b) to 4389f14 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-10T15:57:07Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4225018451-1055-7cj6n 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing alamb/reschedule_io (9124f2b) to 4389f14 (merge-base) diff using: clickbench_partitioned
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-10T16:14:55Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and alamb_reschedule_io
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                  HEAD ┃                   alamb_reschedule_io ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │          1.22 / 4.53 ±6.47 / 17.46 ms │          1.19 / 4.45 ±6.37 / 17.18 ms │     no change │
│ QQuery 1  │        14.46 / 14.86 ±0.21 / 15.06 ms │        11.88 / 12.30 ±0.23 / 12.57 ms │ +1.21x faster │
│ QQuery 2  │        44.00 / 44.40 ±0.29 / 44.74 ms │        36.14 / 36.24 ±0.09 / 36.36 ms │ +1.23x faster │
│ QQuery 3  │        44.07 / 45.93 ±1.48 / 48.56 ms │        33.27 / 34.92 ±0.86 / 35.72 ms │ +1.32x faster │
│ QQuery 4  │    290.34 / 304.66 ±11.29 / 318.33 ms │     259.56 / 266.46 ±6.06 / 275.34 ms │ +1.14x faster │
│ QQuery 5  │     348.81 / 352.99 ±4.94 / 362.55 ms │     302.67 / 307.19 ±3.94 / 312.27 ms │ +1.15x faster │
│ QQuery 6  │          5.94 / 9.54 ±6.29 / 22.09 ms │          7.08 / 9.02 ±2.34 / 13.56 ms │ +1.06x faster │
│ QQuery 7  │        17.12 / 17.61 ±0.51 / 18.51 ms │        13.63 / 14.13 ±0.42 / 14.90 ms │ +1.25x faster │
│ QQuery 8  │    415.75 / 431.67 ±14.78 / 458.48 ms │    368.63 / 381.92 ±12.50 / 404.81 ms │ +1.13x faster │
│ QQuery 9  │    648.78 / 663.68 ±10.81 / 677.34 ms │     530.46 / 535.32 ±4.18 / 540.91 ms │ +1.24x faster │
│ QQuery 10 │       93.53 / 96.64 ±3.56 / 101.84 ms │        77.03 / 78.56 ±2.15 / 82.80 ms │ +1.23x faster │
│ QQuery 11 │     107.55 / 108.67 ±1.00 / 109.88 ms │        86.55 / 87.52 ±1.10 / 89.57 ms │ +1.24x faster │
│ QQuery 12 │     341.49 / 349.66 ±6.53 / 361.43 ms │     297.92 / 301.61 ±4.29 / 309.80 ms │ +1.16x faster │
│ QQuery 13 │    446.82 / 470.22 ±12.99 / 485.08 ms │    419.79 / 437.26 ±12.67 / 451.89 ms │ +1.08x faster │
│ QQuery 14 │     349.69 / 353.00 ±4.54 / 361.99 ms │     305.90 / 310.83 ±2.90 / 314.73 ms │ +1.14x faster │
│ QQuery 15 │    360.26 / 382.10 ±18.53 / 412.60 ms │    326.56 / 345.32 ±21.49 / 383.46 ms │ +1.11x faster │
│ QQuery 16 │    727.21 / 744.14 ±18.60 / 773.96 ms │     686.99 / 691.74 ±5.84 / 700.70 ms │ +1.08x faster │
│ QQuery 17 │     722.76 / 725.25 ±2.38 / 728.92 ms │     677.29 / 685.65 ±5.58 / 691.97 ms │ +1.06x faster │
│ QQuery 18 │ 1447.45 / 1479.35 ±22.76 / 1516.53 ms │ 1374.18 / 1438.15 ±52.85 / 1516.61 ms │     no change │
│ QQuery 19 │        36.46 / 38.71 ±1.83 / 41.73 ms │        31.07 / 31.72 ±0.72 / 33.11 ms │ +1.22x faster │
│ QQuery 20 │    724.72 / 746.23 ±19.77 / 775.14 ms │     517.27 / 522.54 ±5.17 / 531.18 ms │ +1.43x faster │
│ QQuery 21 │     770.94 / 777.53 ±5.53 / 786.95 ms │     597.88 / 601.14 ±2.16 / 603.46 ms │ +1.29x faster │
│ QQuery 22 │  1142.33 / 1148.93 ±5.36 / 1155.94 ms │  1061.05 / 1068.06 ±4.00 / 1072.04 ms │ +1.08x faster │
│ QQuery 23 │  3108.12 / 3118.22 ±8.32 / 3129.96 ms │  3335.10 / 3344.79 ±7.31 / 3353.29 ms │  1.07x slower │
│ QQuery 24 │      98.86 / 103.80 ±3.12 / 108.49 ms │        41.97 / 44.53 ±2.16 / 47.62 ms │ +2.33x faster │
│ QQuery 25 │     140.08 / 140.99 ±0.92 / 142.65 ms │     114.65 / 115.56 ±0.60 / 116.30 ms │ +1.22x faster │
│ QQuery 26 │     101.91 / 104.60 ±1.61 / 106.72 ms │        43.53 / 44.77 ±1.07 / 46.26 ms │ +2.34x faster │
│ QQuery 27 │     855.03 / 860.14 ±5.75 / 871.14 ms │     667.08 / 671.06 ±3.24 / 676.22 ms │ +1.28x faster │
│ QQuery 28 │ 3293.95 / 3315.63 ±12.81 / 3333.56 ms │ 3021.99 / 3047.46 ±20.67 / 3081.71 ms │ +1.09x faster │
│ QQuery 29 │        49.96 / 55.03 ±4.38 / 61.32 ms │        44.23 / 47.72 ±4.78 / 57.11 ms │ +1.15x faster │
│ QQuery 30 │     364.62 / 371.40 ±4.25 / 376.31 ms │     334.58 / 342.38 ±7.19 / 355.55 ms │ +1.08x faster │
│ QQuery 31 │    374.59 / 388.18 ±10.85 / 402.85 ms │     336.75 / 340.35 ±4.54 / 348.85 ms │ +1.14x faster │
│ QQuery 32 │ 1218.93 / 1275.19 ±50.02 / 1366.25 ms │ 1042.23 / 1059.05 ±12.28 / 1077.94 ms │ +1.20x faster │
│ QQuery 33 │ 1549.95 / 1597.95 ±42.29 / 1667.41 ms │ 1499.62 / 1514.00 ±10.63 / 1527.93 ms │ +1.06x faster │
│ QQuery 34 │ 1522.21 / 1558.89 ±29.64 / 1605.74 ms │ 1523.29 / 1567.60 ±37.91 / 1628.51 ms │     no change │
│ QQuery 35 │     397.14 / 403.28 ±5.08 / 411.20 ms │    351.44 / 379.85 ±33.31 / 444.54 ms │ +1.06x faster │
│ QQuery 36 │     116.75 / 121.73 ±3.24 / 125.78 ms │        70.09 / 71.69 ±1.80 / 74.37 ms │ +1.70x faster │
│ QQuery 37 │        48.05 / 49.55 ±1.06 / 50.92 ms │        39.51 / 40.17 ±0.36 / 40.57 ms │ +1.23x faster │
│ QQuery 38 │        74.85 / 76.49 ±1.13 / 78.11 ms │        40.45 / 43.17 ±1.75 / 45.97 ms │ +1.77x faster │
│ QQuery 39 │   217.96 / 276.07 ±102.43 / 480.53 ms │     130.18 / 137.99 ±5.33 / 143.97 ms │ +2.00x faster │
│ QQuery 40 │        24.75 / 25.62 ±1.05 / 27.68 ms │        15.60 / 18.91 ±1.79 / 20.84 ms │ +1.35x faster │
│ QQuery 41 │        20.10 / 21.82 ±1.13 / 23.47 ms │        14.69 / 16.13 ±0.78 / 17.01 ms │ +1.35x faster │
│ QQuery 42 │        20.38 / 20.93 ±0.38 / 21.31 ms │        14.20 / 14.60 ±0.28 / 14.94 ms │ +1.43x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                  ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                  │ 23195.81ms │
│ Total Time (alamb_reschedule_io)   │ 21063.83ms │
│ Average Time (HEAD)                │   539.44ms │
│ Average Time (alamb_reschedule_io) │   489.86ms │
│ Queries Faster                     │         39 │
│ Queries Slower                     │          1 │
│ Queries with No Change             │          3 │
│ Queries with Failure               │          0 │
└────────────────────────────────────┴────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	116.7s
Peak memory	38.0 GiB
Avg memory	29.0 GiB
CPU user	1083.6s
CPU sys	104.1s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	106.0s
Peak memory	40.8 GiB
Avg memory	33.0 GiB
CPU user	1098.4s
CPU sys	84.6s
Peak spill	0 B

File an issue against this benchmark runner

adriangbot · 2026-04-10T16:27:25Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

Comparing HEAD and alamb_reschedule_io
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query     ┃                                   HEAD ┃                   alamb_reschedule_io ┃        Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0  │           1.22 / 4.48 ±6.45 / 17.37 ms │          1.20 / 4.41 ±6.35 / 17.10 ms │     no change │
│ QQuery 1  │ 2193.22 / 2411.64 ±173.89 / 2591.33 ms │ 1857.47 / 1904.61 ±28.98 / 1937.97 ms │ +1.27x faster │
│ QQuery 2  │ 2777.76 / 2889.59 ±120.48 / 3111.81 ms │ 2144.06 / 2237.64 ±58.71 / 2329.18 ms │ +1.29x faster │
│ QQuery 3  │ 2670.49 / 2887.89 ±191.92 / 3142.03 ms │ 2166.44 / 2206.12 ±29.11 / 2253.34 ms │ +1.31x faster │
│ QQuery 4  │  2500.67 / 2643.22 ±94.83 / 2784.16 ms │ 2300.56 / 2358.63 ±34.30 / 2406.29 ms │ +1.12x faster │
│ QQuery 5  │  2717.07 / 2797.11 ±58.43 / 2893.55 ms │ 2384.93 / 2413.89 ±29.95 / 2471.66 ms │ +1.16x faster │
│ QQuery 6  │      204.94 / 210.31 ±6.74 / 221.99 ms │     206.12 / 212.11 ±4.78 / 216.97 ms │     no change │
│ QQuery 7  │ 2204.74 / 2445.52 ±169.27 / 2696.18 ms │ 1912.58 / 1948.97 ±22.37 / 1978.18 ms │ +1.25x faster │
│ QQuery 8  │ 2920.82 / 3053.25 ±109.34 / 3244.93 ms │ 2412.25 / 2505.41 ±49.83 / 2552.71 ms │ +1.22x faster │
│ QQuery 9  │ 2990.71 / 3132.04 ±101.57 / 3287.56 ms │ 2682.19 / 2721.71 ±32.38 / 2779.65 ms │ +1.15x faster │
│ QQuery 10 │ 2613.44 / 2784.66 ±106.60 / 2906.34 ms │ 2229.06 / 2268.40 ±43.12 / 2339.92 ms │ +1.23x faster │
│ QQuery 11 │  2587.17 / 2778.13 ±96.16 / 2845.10 ms │ 2219.76 / 2279.99 ±39.80 / 2322.18 ms │ +1.22x faster │
│ QQuery 12 │ 2614.51 / 2769.07 ±146.35 / 3014.64 ms │ 2448.66 / 2509.57 ±46.01 / 2577.45 ms │ +1.10x faster │
│ QQuery 13 │ 2976.63 / 3091.74 ±106.69 / 3281.10 ms │ 2455.41 / 2536.48 ±45.93 / 2597.60 ms │ +1.22x faster │
│ QQuery 14 │  2952.93 / 3040.13 ±85.73 / 3203.81 ms │ 2397.90 / 2456.21 ±39.85 / 2501.59 ms │ +1.24x faster │
│ QQuery 15 │ 2656.01 / 2787.02 ±171.53 / 3119.13 ms │ 2389.32 / 2429.95 ±27.16 / 2462.31 ms │ +1.15x faster │
│ QQuery 16 │ 3114.83 / 3303.54 ±159.69 / 3509.43 ms │ 2642.44 / 2701.46 ±42.54 / 2760.44 ms │ +1.22x faster │
│ QQuery 17 │  3031.56 / 3106.10 ±60.80 / 3211.60 ms │ 2687.28 / 2702.99 ±15.15 / 2726.00 ms │ +1.15x faster │
│ QQuery 18 │ 3519.06 / 3671.27 ±175.10 / 3967.65 ms │ 3121.36 / 3153.31 ±21.64 / 3185.33 ms │ +1.16x faster │
│ QQuery 19 │ 2136.67 / 2373.14 ±167.80 / 2661.69 ms │ 1772.96 / 1814.94 ±38.40 / 1875.13 ms │ +1.31x faster │
│ QQuery 20 │ 2862.68 / 3105.27 ±136.93 / 3286.39 ms │ 2579.47 / 2674.76 ±52.91 / 2726.55 ms │ +1.16x faster │
│ QQuery 21 │  2904.56 / 3019.34 ±78.55 / 3136.01 ms │ 2764.09 / 2787.64 ±16.19 / 2802.75 ms │ +1.08x faster │
│ QQuery 22 │ 3635.43 / 3830.89 ±171.69 / 4130.70 ms │ 3175.00 / 3225.88 ±48.13 / 3313.83 ms │ +1.19x faster │
│ QQuery 23 │ 4824.27 / 5115.55 ±241.40 / 5435.35 ms │ 5324.30 / 5451.19 ±92.93 / 5606.63 ms │  1.07x slower │
│ QQuery 24 │ 1094.86 / 1421.44 ±195.46 / 1694.39 ms │    499.00 / 531.06 ±33.93 / 593.11 ms │ +2.68x faster │
│ QQuery 25 │ 2623.77 / 2824.09 ±154.27 / 2984.00 ms │ 2235.47 / 2310.50 ±40.01 / 2349.00 ms │ +1.22x faster │
│ QQuery 26 │  1349.33 / 1413.30 ±53.70 / 1493.57 ms │    497.72 / 595.43 ±66.18 / 667.73 ms │ +2.37x faster │
│ QQuery 27 │ 3078.80 / 3310.47 ±205.46 / 3618.19 ms │  2824.18 / 2834.62 ±6.10 / 2839.96 ms │ +1.17x faster │
│ QQuery 28 │ 5366.35 / 5554.94 ±131.96 / 5703.44 ms │ 5062.32 / 5091.28 ±30.94 / 5143.79 ms │ +1.09x faster │
│ QQuery 29 │ 2608.33 / 2800.28 ±202.93 / 3173.63 ms │ 2206.54 / 2268.25 ±37.31 / 2317.65 ms │ +1.23x faster │
│ QQuery 30 │ 2708.26 / 3041.53 ±246.89 / 3455.11 ms │  2484.60 / 2499.22 ±9.83 / 2515.40 ms │ +1.22x faster │
│ QQuery 31 │ 2646.37 / 2892.19 ±163.21 / 3084.02 ms │ 2398.69 / 2449.10 ±38.26 / 2504.26 ms │ +1.18x faster │
│ QQuery 32 │ 2841.87 / 3057.58 ±133.86 / 3256.41 ms │ 2563.61 / 2617.12 ±33.26 / 2665.98 ms │ +1.17x faster │
│ QQuery 33 │ 3548.73 / 3865.56 ±214.44 / 4051.83 ms │ 3302.96 / 3341.68 ±19.69 / 3357.14 ms │ +1.16x faster │
│ QQuery 34 │  3613.04 / 3754.65 ±96.62 / 3868.58 ms │ 3369.81 / 3396.79 ±19.36 / 3419.38 ms │ +1.11x faster │
│ QQuery 35 │ 2825.45 / 2913.78 ±100.43 / 3108.22 ms │ 2408.26 / 2432.31 ±13.41 / 2448.42 ms │ +1.20x faster │
│ QQuery 36 │     216.07 / 327.56 ±75.95 / 417.79 ms │    192.96 / 276.61 ±52.07 / 341.75 ms │ +1.18x faster │
│ QQuery 37 │     189.29 / 246.39 ±45.80 / 313.17 ms │    150.70 / 238.08 ±54.75 / 308.55 ms │     no change │
│ QQuery 38 │     210.43 / 265.47 ±51.64 / 339.15 ms │    175.18 / 262.39 ±55.08 / 334.00 ms │     no change │
│ QQuery 39 │     304.82 / 399.44 ±68.66 / 487.79 ms │    236.13 / 316.00 ±51.69 / 382.17 ms │ +1.26x faster │
│ QQuery 40 │     166.09 / 223.11 ±46.27 / 291.47 ms │    148.34 / 234.45 ±53.38 / 302.66 ms │  1.05x slower │
│ QQuery 41 │     159.27 / 218.24 ±45.64 / 284.31 ms │    144.54 / 229.88 ±53.84 / 299.18 ms │  1.05x slower │
│ QQuery 42 │     158.76 / 216.88 ±44.55 / 281.65 ms │    144.59 / 196.62 ±35.56 / 239.63 ms │ +1.10x faster │
└───────────┴────────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                  ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                  │ 105997.79ms │
│ Total Time (alamb_reschedule_io)   │  89627.67ms │
│ Average Time (HEAD)                │   2465.06ms │
│ Average Time (alamb_reschedule_io) │   2084.36ms │
│ Queries Faster                     │          36 │
│ Queries Slower                     │           3 │
│ Queries with No Change             │           4 │
│ Queries with Failure               │           0 │
└────────────────────────────────────┴─────────────┘

Resource Usage

clickbench_partitioned — base (merge-base)

Metric	Value
Wall time	531.2s
Peak memory	35.4 GiB
Avg memory	26.5 GiB
CPU user	1039.2s
CPU sys	117.2s
Peak spill	0 B

clickbench_partitioned — branch

Metric	Value
Wall time	449.3s
Peak memory	36.7 GiB
Avg memory	27.4 GiB
CPU user	1053.4s
CPU sys	115.0s
Peak spill	0 B

File an issue against this benchmark runner

alamb · 2026-04-10T16:28:52Z

😎

│ Average Time (HEAD)                │   2465.06ms │
│ Average Time (alamb_reschedule_io) │   2084.36ms │

│ Average Time (HEAD)                │   539.44ms │
│ Average Time (alamb_reschedule_io) │   489.86ms │

Ok, I am going to stop polishing this and focus on getting it in asap

github-actions bot added the datasource Changes to the datasource crate label Apr 3, 2026

alamb force-pushed the alamb/reschedule_io branch 2 times, most recently from 9789b9f to d5c4c5f Compare April 6, 2026 11:04

This comment has been minimized.

Sign in to view

alamb mentioned this pull request Apr 6, 2026

Rewrite FileStream in terms of Morsel API #21342

Open

This comment has been minimized.

Sign in to view

This was referenced Apr 6, 2026

Introduce Morselizer API, rewrite ParquetOpener to ParquetMorselizer #21327

Merged

Sketch out a Morselize API #20820

Closed

Introduce morsel-driven Parquet scan #20529

Open

alamb added 2 commits April 9, 2026 12:59

Rewrite FileStream in terms of Morselize API

8cd86f8

Address feedback from @adriangb

393c03f

alamb force-pushed the alamb/reschedule_io branch from 1649c65 to acde88a Compare April 9, 2026 18:24

This comment has been minimized.

Sign in to view

alamb added 4 commits April 10, 2026 11:22

Merge remote-tracking branch 'apache/main' into alamb/file_stream_split

9923ed9

Add Dynamic work scheduling in FileStream

97bc9c9

Revert "simple"

416570e

This reverts commit c479ed68e02a4337eb9cbafa8ec1d70b4e5c2345.

Refactor shared state creation

74032d6

alamb force-pushed the alamb/reschedule_io branch from 8a6dc84 to 74032d6 Compare April 10, 2026 15:23

spelling

9124f2b

Conversation

alamb commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Dandandan commented Apr 6, 2026

Uh oh!

This comment has been minimized.

alamb commented Apr 6, 2026

Uh oh!

alamb commented Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Dandandan commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Apr 10, 2026

Uh oh!

alamb commented Apr 10, 2026

Uh oh!

alamb commented Apr 10, 2026

Uh oh!

adriangbot commented Apr 10, 2026

Uh oh!

adriangbot commented Apr 10, 2026

Uh oh!

adriangbot commented Apr 10, 2026

Uh oh!

adriangbot commented Apr 10, 2026

Uh oh!

alamb commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alamb commented Apr 3, 2026 •

edited

Loading

alamb commented Apr 6, 2026 •

edited

Loading

Dandandan commented Apr 10, 2026 •

edited

Loading

alamb commented Apr 10, 2026 •

edited

Loading